Skip to content

Add deployment-managed core migration primitives - #2437

Merged
ascorbic merged 9 commits into
mainfrom
codex/issue-2276-migrations-core
Aug 18, 2026
Merged

Add deployment-managed core migration primitives#2437
ascorbic merged 9 commits into
mainfrom
codex/issue-2276-migrations-core

Conversation

@ascorbic

@ascorbic ascorbic commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds the core primitives for deployment-managed migrations: migration policies, manifests, configuration loading, direct SQLite/libSQL/PostgreSQL executors, runtime enforcement, and package exports. It also adds upgrade, policy, manifest, adapter, and executor coverage.

Part of #2276

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change)
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable). Do not include messages.po changes except in translation PRs — a workflow extracts catalogs on merge to main.
  • I have added a changeset (if this PR changes a published package)
  • New features link to an approved Discussion: RFC: Deployment-managed core migrations #2418

Approved design discussion: #2418. Package typechecking, quick lint, and targeted formatting pass. The core suite passes 5,418 tests; its remaining failure is the pre-existing macOS /var versus /private/var path-normalization assertion.

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: GPT-5 Codex

Screenshots / test output

Non-visual change. Core package build and typecheck pass, query-count snapshots are unchanged, and the relevant migration tests pass.

@changeset-bot

changeset-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f5403a7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
emdash Minor
@emdash-cms/cloudflare Minor
@emdash-cms/sandbox-workerd Patch
@emdash-cms/plugin-mcp-smoke Major
@emdash-cms/fixture-perf-site Patch
@emdash-cms/perf-demo-site Patch
@emdash-cms/cache-demo-site Patch
@emdash-cms/do-demo-site Patch
@emdash-cms/do-solo-demo-site Patch
@emdash-cms/admin Minor
@emdash-cms/auth Minor
@emdash-cms/blocks Minor
@emdash-cms/gutenberg-to-portable-text Minor
@emdash-cms/x402 Minor
create-emdash Minor
@emdash-cms/auth-atproto Patch
@emdash-cms/plugin-embeds Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
docs e450006 Aug 18 2026, 01:17 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
emdash-demo-cache f5403a7 Aug 18 2026, 02:25 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
emdash-demo-do f5403a7 Aug 18 2026, 02:29 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
emdash-playground 0489cd4 Aug 13 2026, 11:27 AM

@github-actions

Copy link
Copy Markdown
Contributor

Scope check

This PR changes 3,358 lines across 51 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

Comment thread packages/core/tests/unit/migrations/config-loader.test.ts Dismissed
@pkg-pr-new

pkg-pr-new Bot commented Aug 12, 2026

Copy link
Copy Markdown

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2437

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2437

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2437

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2437

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2437

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2437

emdash

npm i https://pkg.pr.new/emdash@2437

create-emdash

npm i https://pkg.pr.new/create-emdash@2437

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2437

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2437

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2437

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2437

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2437

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2437

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2437

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2437

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2437

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2437

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2437

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2437

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2437

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2437

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2437

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2437

commit: f5403a7

@ascorbic
ascorbic marked this pull request as ready for review August 12, 2026 16:32
Copilot AI lite review requested due to automatic review settings August 12, 2026 16:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the core building blocks for deployment-managed core database migrations in packages/core, including a runtime policy (auto/check/manual), a validated secret-free migration manifest written at build/sync time, direct migration executors for SQLite/libSQL/Postgres, and public exports (identity + exact status) intended for deployment tooling.

Changes:

  • Introduces runtime migration modes and enforces them during runtime initialization + middleware handling.
  • Adds migration manifest primitives (identity, manifest schema/validation, manifest writer, config loader, integration metadata) and writes .emdash/migrations.json during astro build/astro sync.
  • Adds direct migration executors for SQLite/libSQL/Postgres plus a new exact migration status API (knownApplied/pending/unknownApplied) with broad test coverage.

Reviewed changes

Copilot reviewed 50 out of 51 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/core/tsdown.config.ts Adds new migration-related entrypoints to the build.
packages/core/tests/utils/test-db.ts Exposes exact migration status helper for dialect tests.
packages/core/tests/unit/migrations/manifest.test.ts Unit coverage for manifest build + validation (schema, secrets, entrypoints).
packages/core/tests/unit/migrations/manifest-writer.test.ts Unit coverage for atomic manifest writes and failure cleanup.
packages/core/tests/unit/migrations/identity.test.ts Unit coverage for identity fingerprinting and immutability.
packages/core/tests/unit/migrations/direct-executor.test.ts Unit coverage for direct executor behavior, lifecycle, and failure modes.
packages/core/tests/unit/migrations/config-loader.test.ts Unit coverage for config discovery/evaluation + metadata extraction + i18n normalization.
packages/core/tests/unit/i18n/normalize.test.ts Unit coverage for Astro i18n normalization helper.
packages/core/tests/unit/db/migration-executors.test.ts Unit coverage for sqlite/libsql/postgres migration executor modules.
packages/core/tests/unit/db/migration-adapters.test.ts Unit coverage for adapter migration descriptors and secret separation.
packages/core/tests/unit/database/migration-policy.test.ts Unit coverage for migration config normalization + runtime enforcement query counts.
packages/core/tests/unit/astro/setup-dev-bypass.test.ts Ensures dev-bypass/setup routes don’t run migrations directly.
packages/core/tests/unit/astro/middleware-migration-policy.test.ts Ensures middleware returns generic 503 behavior on pending migrations/manual mode.
packages/core/tests/unit/astro/integration/migration-policy.test.ts Ensures integration rejects invalid migration policy modes.
packages/core/tests/unit/astro/integration/migration-manifest.test.ts Ensures build/sync write secret-free manifests and metadata is attached correctly.
packages/core/tests/integration/runtime/create.test.ts Ensures check mode doesn’t trigger migration-failure backoff.
packages/core/tests/integration/database/migrations.test.ts Adds integration coverage for exact migration status behavior + query counts.
packages/core/tests/integration/database/dialect-compat.test.ts Ensures exact migration status works across dialects.
packages/core/src/virtual-modules.d.ts Extends virtual config typing to include migrations config.
packages/core/src/migrations/target.ts Adds safe target fingerprinting + env var resolution helper.
packages/core/src/migrations/protocol.ts Defines executor protocol types (request/target/report/factory).
packages/core/src/migrations/manifest.ts Adds manifest v1 schema, secret-free validation, and serialization.
packages/core/src/migrations/manifest-writer.ts Writes validated manifest atomically to .emdash/migrations.json.
packages/core/src/migrations/manifest-builder.ts Builds a manifest from identity + i18n + adapter migration metadata.
packages/core/src/migrations/integration-metadata.ts Attaches secret-free migration metadata to the Astro integration (symbol-keyed).
packages/core/src/migrations/index.ts Exports migration identity, protocol, and direct executor APIs.
packages/core/src/migrations/identity.ts Adds core migration identity + fingerprint generation.
packages/core/src/migrations/direct-executor.ts Implements single-use direct executor (check/apply) with i18n scoping.
packages/core/src/migrations/config-loader.ts Adds config-file discovery and safe evaluation fallback to build manifests from config.
packages/core/src/index.ts Exports exact migration status + MIGRATION_NAMES/types from the public entry.
packages/core/src/i18n/normalize.ts Adds normalization of Astro i18n input into EmDash’s I18nConfig.
packages/core/src/emdash-runtime.ts Enforces runtime migration policy during DB init; avoids backoff for pending-migration errors.
packages/core/src/db/sqlite-migrations.ts Adds deployment executor module for SQLite migrations.
packages/core/src/db/postgres-migrations.ts Adds deployment executor module for Postgres migrations.
packages/core/src/db/libsql-migrations.ts Adds deployment executor module for libSQL migrations.
packages/core/src/db/index.ts Exposes exact migration status and MIGRATION_NAMES via db exports.
packages/core/src/db/adapters.ts Adds adapter-provided migration metadata (secret-free manifestConfig).
packages/core/src/database/migrations/runner.ts Adds MIGRATION_NAMES + exact migration status query helper.
packages/core/src/database/migrations/policy.ts Adds migration config normalization + runtime enforcement for auto/check/manual.
packages/core/src/database/index.ts Exposes exact migration status and MIGRATION_NAMES from database exports.
packages/core/src/astro/routes/api/setup/index.ts Removes direct migration running from setup route.
packages/core/src/astro/routes/api/setup/dev-bypass.ts Removes direct migration running from dev-bypass route (runtime now governs).
packages/core/src/astro/routes/api/auth/dev-bypass.ts Removes direct migration running from auth dev-bypass route.
packages/core/src/astro/middleware.ts Resolves effective migration mode and returns 503s for pending/manual-missing-schema cases.
packages/core/src/astro/integration/runtime.ts Adds migrations config to EmDash integration config type.
packages/core/src/astro/integration/index.ts Normalizes migration config; writes migration manifest during build/sync; attaches migration metadata.
packages/core/package.json Exposes new public subpath exports for migration executors and ./migrations.
.changeset/runtime-migration-policy.md Changeset for runtime migration policy modes.
.changeset/direct-migration-adapters.md Changeset for deployment migration executors on built-in adapters.
.changeset/deployment-migration-primitives.md Changeset for public migration identity + exact status APIs.
.changeset/deployment-migration-manifest.md Changeset for manifest emission during build/sync.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/core/src/migrations/manifest.ts Outdated

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR introduces a well-scoped set of deployment-managed migration primitives: a runtime migration policy (auto/check/manual), build-time manifest generation and writing, identity/fingerprinting, and direct SQLite/libSQL/PostgreSQL executors. The approach fits EmDash’s architecture—writer runs at Astro build/sync time, executors are loaded from the same published subpaths used by the runtime, and the runtime enforces the policy without adding queries to the logged-out hot path beyond the existing migration fast-path count.

I read the new migration source, the adapter migration entrypoints, the integration wiring, the runtime policy integration, the Virtual-module type updates, the package exports, tsdown entries, and the test coverage. The code is generally clean, uses Kysely for SQL safety, keeps credentials out of the build artifact, and adds real behavioral tests for check/apply, secret rejection, target fingerprinting, config loading, and manual-mode middleware behavior.

One real logic bug stands out: the direct executor discards a successful migration report if closing the dialect fails. I also have one minor testing nit about a hard-coded SHA fingerprint literal.

Comment thread packages/core/src/migrations/direct-executor.ts Outdated
Comment thread packages/core/tests/unit/migrations/identity.test.ts Outdated
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond needs-rebase overlap labels Aug 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

@ascorbic
ascorbic force-pushed the codex/issue-2276-migrations-core branch from 96ba1b5 to 4ccb0d3 Compare August 13, 2026 08:20
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed needs-rebase review/awaiting-author Reviewed; waiting on the author to respond labels Aug 13, 2026
@ascorbic ascorbic added the bot:review Trigger an emdashbot code review on this PR label Aug 13, 2026

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a well-scoped set of deployment-managed migration primitives: a runtime migration policy (auto/check/manual), build-time manifest generation and writing, migration identity/fingerprinting, and direct SQLite/libSQL/PostgreSQL executors loaded from the same published subpaths the runtime uses. The approach fits EmDash’s architecture—writer runs at Astro build/sync time, the runtime enforces the policy without new logged-out hot-path queries beyond the existing migration fast-path count, and credentials are kept out of the build artifact. It links to the approved Discussion (#2418) and includes changesets, tests, and the necessary package/tsdown entries.

I read the new migration source, the adapter entrypoints, the integration and middleware wiring, the runtime policy integration, the Virtual-module type updates, the package exports, tsdown entries, and the test coverage. This is a static review; I did not run the test suite or build.

Prior-review status:

  • Fixed: the direct executor no longer discards a successful migration report when closing the dialect fails; it now returns the report while logging the close error.
  • 🔁 Still open: the hard-coded SHA-256 fingerprint literal in identity.test.ts remains.

New finding: the middleware’s 503 response tells operators to run emdash migrate, but the EmDash CLI does not expose a migrate subcommand yet. That guidance is currently unfulfilled and will confuse operators when check or manual mode blocks traffic.

Verdict: comment — one real needs_fixing issue (the CLI message), one testing suggestion, and otherwise a clean, deliberate implementation.

Comment thread packages/core/src/astro/middleware.ts Outdated
Comment thread packages/core/tests/unit/migrations/identity.test.ts Outdated
@emdashbot emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Aug 13, 2026
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Aug 13, 2026
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Aug 13, 2026
@ascorbic
ascorbic force-pushed the codex/issue-2276-migrations-core branch from 0489cd4 to e450006 Compare August 18, 2026 13:08
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond needs-rebase labels Aug 18, 2026
@ascorbic ascorbic added the bot:review Trigger an emdashbot code review on this PR label Aug 18, 2026

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a re-review of the deployment-managed migration primitives PR (#2437). The implementation remains the right shape for #2276/#2418: the build-time manifest keeps secrets out of the artifact, auto/check/manual modes are wired into the existing runtime and middleware path without adding logged-out hot-path queries, and the direct SQLite/libSQL/PostgreSQL executors reuse the same published adapter subpaths as production.

I re-read the migration identity, manifest builder/writer, direct executor, policy, integration wiring, runtime policy integration, middleware 503 paths, and the new/updated tests. Conventions still look good: SQL is parameterized/identifier-safe, the manifest secret-free validator rejects credentials and connection URLs, the setup probe is now gated to auto mode so the query-count snapshot stays unchanged, changesets are present and user-facing, and no admin UI strings were added.

The only unresolved issue from the previous review is still present: the fingerprint test hard-codes a SHA-256 literal. That assertion is a config-pin test that restates the implementation and cannot fail on any real regression other than an intentional algorithm change. The preceding assertions already cover determinism (expect(second).toBe(first)) and hex formatting (expect(first).toMatch(/^[0-9a-f]{64}$/)), so the literal should be removed.

No new security, data-integrity, or logged-out query regressions were introduced.

Comment thread packages/core/tests/unit/migrations/identity.test.ts Outdated
@emdashbot emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Aug 18, 2026
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Aug 18, 2026
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond labels Aug 18, 2026
@ascorbic
ascorbic merged commit 34336e1 into main Aug 18, 2026
54 checks passed
@ascorbic
ascorbic deleted the codex/issue-2276-migrations-core branch August 18, 2026 15:16
@emdashbot emdashbot Bot mentioned this pull request Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core overlap review/needs-rereview Author pushed changes since the last review size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants